Safe from “harm”: The governance of violence by platforms

A number of issues have emerged related to how platforms moderate and mitigate “harm.” Although recently developed more explicit policies in regard what constitutes “hate speech” “harmful content,” it appears that often use subjective judgments harm specifically pertains spectacular, physical violence—but takes on many shapes complex forms. The politics defining “harm” “violence” within these are dynamic, represent entrenched histories control over definitions extends people's perceptions them. Via a critical discourse analysis policy documents from three major (Facebook, Twitter, YouTube), we argue platforms' narrow violence not just insufficient but result engaging form symbolic violence. Moreover, the position as floating signifier, imposing conceptions is manifests, who impacts. Rather than changing mechanisms their design enable harm, reconfigure intentionality causality try stop users being “harmful,” which, ironically, perpetuates harm. We provide suggestions, namely restorative justice-focused approach, addressing platform 出现了许多与平台如何缓和和减轻“伤害”有关的问题。尽管平台最近针对“仇恨言论”和“有害内容”的构成制定了更明确的政策，但似乎平台经常使用与壮观的身体暴力特别相关的主观伤害判断——但伤害具有多种形式和复杂性形式。在这些平台中定义“伤害”和“暴力”的政治是复杂和动态的，并且代表了对这些定义的控制如何延伸到人们对它们的看法的根深蒂固的历史。通过对来自三个主要平台（Facebook、Twitter 和 YouTube）的政策文件的批判性话语分析，我们认为平台对伤害和暴力的狭隘定义不仅不够，而且导致这些平台参与了一种象征性的暴力形式。此外，平台将伤害定位为一个浮动的能指，不仅强加关于暴力是什么以及它如何表现的概念，还强加了它影响的人的概念。这些平台并没有改变造成伤害的设计机制，而是重新配置意图和因果关系，以试图阻止用户“有害”，具有讽刺意味的是，这会使伤害永久化。我们提供了一些建议，即以恢复性司法为重点的方法来解决平台损害。 Han surgido varios problemas relacionados con la forma en que las plataformas moderan y mitigan el "daño". Aunque han desarrollado recientemente políticas más explícitas respecto lo constituye "discurso de odio" "contenido dañino", parece menudo usan juicios subjetivos daño se relacionan específicamente violencia física espectacular, pero toma muchas formas complejos. formularios Las definición “daño” “violencia” dentro estas son complejas dinámicas, representan historias arraigadas cómo sobre definiciones extiende percepciones gente tiene ellas. través un análisis crítico del discurso los documentos tres principales Twitter argumentamos limitadas no solo insuficientes sino dan como resultado involucren una simbólica. Además, posicionan significante flotante, imponiendo conceptos qué es manifiesta, también quién impacta. En lugar cambiar mecanismos su diseño permiten daño, reconfiguran intencionalidad causalidad para tratar evitar usuarios sean "perjudiciales", que, irónicamente, perpetúa daño. Brindamos serie sugerencias, saber, enfoque centrado justicia restaurativa, abordar plataforma. Terms service moderation always existed some degree, particularly when came illegal content like child sexual exploitation or sale regulated goods (drugs, firearms, etc.). But recent years, scrambled adjust around content” following tragedies involving terrorism, mass shootings, other acts could be traced affordances (Daniels, 2018; Odag et al., 2019). Indeed, after years attempting absolve themselves blame responsibility for occurs services, Facebook, Reddit, YouTube, others implemented new sweeping banned groups, hashtags, means digital connection coordination afforded (Donovan, 2020a; Ganesh, Gillespie, Suzor These had reckon with role radicalization, extremist organizing, rampant disinformation threatens social fabric our society; particular, COVID-19 pandemic, violent white supremacist organizing during Black Lives Matter protests summer 2020, January 6 U.S. Capitol Insurrection. repertoires action adopt response problems—what has been called governance by (Gillespie, 2017)—exhibit variety pertaining commercial interests, isomorphic decisionmaking, influence responsibilities, sovereignty decisions about behavior worthy remediation. In terms must keep engaged, which ensuring popular accounts remain active visible. This led inequitable practices, wherein high-profile receive greater leniency complete immunity comes enforcing rules (Caplan & 2020; Horwitz, 2021). Platforms also balance concerns those advertisers. However, direct generators revenue, advertisers' relation practices tend hold sway average users. Beyond user concerns, develop lean AI automation identifying managing harmful minimize costs associated human moderation. Yet, technologies frequently fail at detecting severe instances nuanced (e.g., hate speech; Gorwa 2020). Platform exhibit isomorphism Danah, 2018), similar changes simultaneously one another. suggests decision-making only public pressure peer pressure. can see such in, example, remove figures conspiracy theorist Alex Jones, former president Donald Trump, identified central “harmful” communities engage offline (Fischer Gold, 2021; Schwartz, coevolution helps urge particular normative visions they should intervene post share based certain framings “violent” enough warrant removal outright bans. As learn each other, strides made updating speech, content, misinformation, forms malicious content. Neverthless, pertain Such belie political concepts processes subsumed (Arendt, 1970; Bourdieu, 1999; Giroux, 2017). circumscribing “harm,” “violence,” “danger” dynamic extend significance severity. Through policies, delimit share, reinforce ideas frames behavior. They govern both discursive material expressions this governance, powerfully shape notions violence, effectively actions directing users' understanding not. purpose study examine media YouTube) understand ways conceptualize, implement, enforce lenses Bourdieu's (1999) framework Gillespie's conception 2017), conduct YouTube define police milieu. Specifically, turn attention publicly accessible conceptualizing via machine-driven interventions. By open coding substantively address examining underlying meaning hidden language (Fairclough, 1992, 2014; van Dijk, 1993), tease out recurrent contours classifying With analysis, suggest danger insufficient, ideological hegemony, impacts mechanisms. find rather through an attempt Following aforementioned events briefly touched upon introduction, civic process received increased scrutiny (Gorwa, interventions sought Europe, United States, countries issues, tense relationship between external forces internal come into full view (Flew 2019; Ganesh Bright, 2018). initially portrayed even marketed beneficial democratic technologies, current political, social, cultural situation points having deleterious effects all spheres Hao, Hemsley Of course, platform, person, society sociotechnical one, influences influenced governance. How governed fraught debate, term “platform” itself vague ambiguous allows corporations flexibility societal roles responsibilities 2010). Governance, then, “less set capacity,” 2019, p. 856) aligned ability make For online services predated them, reflected prevent abuse enforcement bans censorship. landscape, algorithms, interfaces, well Plantin individual level, actors inform own regulatory frameworks—making “specific network interactions spanning different behaviours” includes government, academics, nongovernmental journalists, stakeholders (Dijck enmeshed global politics—intervening every aspect ordinary life 2017; Punathambekar, becomes issue consider themselves, leading phenomenon Gillespie noted “governance platforms” (emphasis added; importance dynamics then outward participation initiatives Global Internet Forum Counter Terrorism (GIFCT) understandably gives scholars pause—these self-governance, uses (Duguay, 2016; Gorwa, Myers West, resulted perfunctory adherence belief self govern, referred “techlash” (a portmanteau “technology” “backlash”) global, uneven deeply imbued discriminatory beliefs regarding gender, race, ability, sexuality, gender expression (Gerrard Thornham, Gray Zolides, Adding complexity, much relies underpaid contractors (many South) (Roberts, 2019) algorithmic due sheer “scale” shared daily Governance entails gendered, racialized, sexualized very people meant protect Hiding behind “free justification “platform values,” neoliberal interpretations freedom speech (Hokka, 2021) claim practice “neutrality” (Hallinan addition, investigative reports leaked memos employees point expression” values best hypocritical worst manipulative negligent. revealed Facebook showed benefit “freer” essentially whitelisted (Horwitz, Earlier reporting similarly documented inconsistency YouTube's policies—for time, instructing moderators take down videos depicting drug-related Mexico, conflicts Syria Russia (Buni Chemaly, 2016). begs question purposes serves (or lack action) actively trying harm? become involved activities everyday life, discussions need “cooperative responsibility” platforms, governments, model where falls (Helberger transnational corporations, driven private strictly claims providing Helberger key factor self-directed frameworks. Platforms' varying curtail reveal inherent relying themselves. may visible mis/disinformation manipulation. Companies approaches toward moderating content: artisanal (small scale, manual), community-reliant (relying report content), industrial (large automated), combination (Caplan, amount posted, clear violent/harmful crucial equal process. complicated factors context, languages, moderators' inevitable subjectivity Hate defined crime laws Western legal frameworks, damaging exacerbate help alleviate systemic caused first place (Schoenebeck Blackwell, Online however, complex, taking multiple forms, Schoenbeck Blackwell limited psychological distress, oppression marginalization, threats free (2021). Harassment, categories considered (Banko seem straightforward, seems “objectively” harmful, terrorist abstract fly under radar coded maneuvering censorship (Gerrard, Murthy, Harassment most highly contested despite attempts “clearly” harassment and/or (Pohjonen, difficult define. Evidencing its harms way meets burden proof significant problem plagues conceptualization (Gelber McNamara, Similarly, long struggled meaningfully (Gray, (2021) noted, focus tends towards removing violate prioritizes retribution actual structural recognize repair still persists companies' harassment, kinds typologizing occur necessary Banks, 2010), do leads place. assign quantitatively lead overlooking allowing persist if meet numeric threshold, notion prioritized (Scheuerman Despite implementing continuing update kind inconsistencies abound across them (Pater Expecting prevalent previous section frameworks evidence circulates harms, largely left self-regulate, doing so any government (Benesch, 2020), without knowledge (Kalsnes Ihlbæk, Due extensive reach scale moderate, whether governable valid (Ganesh, judging cases, inaction, decide States headquartered, globally. perpetuate Symbolic refers power, hierarchies, inequalities maintained less force domination, (Bourdieu, Morgan Björkert, 2006). better legitimate presented documents, because imposed perceive “real” “valid” (Gray powerful insidious invisible, internalized silent (Morgan Social medium discourse—they systems dictating said, traditional communication channels (Recuero, 2015). It argued consistent self-regulation dominate globally opaque unseen 2020b; way, act gatekeepers audiences determining appropriate inappropriate Platforms, construct reality favor views modes power others, perpetuating -isms world Recuero, incorporating Galtung “those aspects culture, sphere existence … used justify legitimize violence,” (Galtung, 1990, 291). (1990) notes, violence—similarly violence—can look feel natural. Cultural contributes fall range conventional understandings “changing moral color red/wrong green/right least yellow/acceptable” 292). Simply put, helpful additional nature us go beyond affect lived experiences offline. rest article, conceptualize Further, formations—as actions—perpetuate serve legitimizing collected data put forth (Twitter, YouTube). keyword searches variants hate, etc.), assembled corpus 106 (22 36 48 All were March 2021 July 2021. important note retrieved Insurrection, among ramifications. Nonetheless, dated, offered “last updated” date fell 2 years.1 constant state flux, believe reflects fundamental guide (in)actions. included list Appendix A, refer specific assigned findings discussion. Using Galtung's concept conducted 1993) Per recommendations Lazar conducting feminist paid close asymmetries present (Lazar, 2007). Since interested “violent/harmful (defined broadly themselves) merely governing acts. Discursive acts, reproduce relations follow Fairclough's (2014) analy object (the documents) objects viewed perceived sociohistorical conditions 2014). To analyze began independently reading subset writing analytic initial observations. this, focused references variant (violence, met discuss From discussion synthesized codes capture descriptive observed, harm–namely, presuppositions embedded linguistic representations documents. After codes, cycle higher-level illuminate found none definition Instead, described types examples instance, provides “physical,” “psychological,” “informational” [TW01]. characteristics actions, including “potential incite violence” “impact bullying/harassment.” offers narrowest description seeming limit potential “physical harm/violence,” “death,” “inciting hostility.” Table 1, brief descriptions observations note, While elusive, did patterned emphasis concept—namely, safety, cyberbullying, (i.e., pornography), meaningful enagement sites. framework, things copyright, (firearms, drugs, defamation (in case “the process” was likely manipulated sow discord elections worldwide (Benkler Woolley Howard, treated discrete concept, flexibly qualified relevant offenses. result, mold according deem occurred. signifiers. According Laclau, explained Farkas Schou, signifier “a fundamentally opposing projects constructing identities, antagonisms,” (Farkas 2005, 300). avoiding stable speech,” “violent carefully navigate invoked weighty terms, while instilling hegemonic ills. And signifiers, struggles gets it—in essence, absorbed matrix domination language, framework. deserving protection enforcement, reify order. sticking “fixed” constitutive seemed t